Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support proxy_url and ssl_ca_certs options for gRPC #341

Merged
merged 1 commit into from
May 13, 2024
Merged

Support proxy_url and ssl_ca_certs options for gRPC #341

merged 1 commit into from
May 13, 2024

Conversation

daverigby
Copy link
Contributor

@daverigby daverigby commented May 10, 2024 •
edited
Loading

Problem

The PineconeGRPC client does not currently honor the proxy_url and ssl_ca_certs arguments - they are accepted but silently ignored.

This means the gRPC client cannot be used in enviroments which require a proxy to access the Pinecone service.

Solution

Address by wiring up these two config options to the corresponding options in the underlying grpc library - in addition to them still being passed to the HTTP-based control-plane connections.

Usage is very similar to the non-gRPC client - for example:

pc = PineconeGRPC(my_api_key,
                  proxy_url="http://localhost:8080",
                  ssl_ca_certs="/path/to/my/ca-cert.pem")

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Test Plan

Proxy tests expanded to also test PineconeGRPC

@daverigby
Support proxy_url and ssl_ca_certs options for gRPC
2380c6a 
The PineconeGRPC client does not currently honor the proxy_url and
ssl_ca_certs arguments - they are accepted but silently ignored.

This means the gRPC client cannot be used in enviroments which require
a proxy to access the Pinecone service.

Address by wiring up these two config options to the corresponding
options in the underlying grpc library - in addition to them still
being passed to the HTTP-based control-plane connections.

Usage is very similar to the non-gRPC client - for example:

    pc = PineconeGRPC(my_api_key,
                      proxy_url="http://localhost:8080",
                      ssl_ca_certs="/path/to/my/ca-cert.pem")
@daverigby daverigby force-pushed the daver/grpc_proxy_ssl_ca_cert branch from ad94eee to 2380c6a Compare May 10, 2024 21:05
@daverigby daverigby marked this pull request as ready for review May 10, 2024 21:08
@daverigby
Copy link
Contributor Author

I note that the proxy tests are not currently run via CI. I re-ran them manually (for USE_GRPC=false and true) on Amazon Linux 2023.

In non-GRPC mode all tests pass (as before), in GRPC-mode all tests which are supported pass (gRPC doesn't support https:// proxies or proxy auth):

$ USE_GRPC=true PINECONE_API_KEY=f0ae7459-aa9c-46c2-8869-ff115dc15901 PINECONE_INDEX_NAME=test poetry run pytest tests/integration/proxy_config -s -v 
====================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.9.16, pytest-8.0.0, pluggy-1.3.0 -- /data/daver/repos/pinecone-python-client/.venv/bin/python
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /data/daver/repos/pinecone-python-client
plugins: timeout-2.2.0, asyncio-0.15.1, mock-3.6.1, cov-2.10.1, benchmark-4.0.0
collected 6 items                                                                                                                                                                                                                                               

tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_https_proxy_with_self_signed_cert SKIPPED (gRPC doesn't support 'https://' proxy URLs)
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_http_proxy_with_self_signed_cert docker run -d --rm --name proxy1 -p 8080:8080 -v /data/daver/repos/pinecone-python-client/tests/integration/proxy_config/.mitm/proxy1:/home/mitmproxy/.mitmproxy mitmproxy/mitmproxy mitmdump
docker run -d --rm --name proxy2 -p 8081:8080 -v /data/daver/repos/pinecone-python-client/tests/integration/proxy_config/.mitm/proxy2:/home/mitmproxy/.mitmproxy mitmproxy/mitmproxy mitmdump --set proxyauth=testuser:testpassword
PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_with_ssl_verification_disabled_emits_warning SKIPPED (gRPC doesn't support disabling ssl_verify)
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_with_incorrect_cert_path PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_with_valid_path_to_incorrect_cert PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_that_requires_proxyauth SKIPPED (gRPC doesn't support proxy auth)

================================================================================================================= 3 passed, 3 skipped in 8.21s ==================================================================================================================

$ USE_GRPC=false PINECONE_API_KEY=f0ae7459-aa9c-46c2-8869-ff115dc15901 PINECONE_INDEX_NAME=test poetry run pytest tests/integration/proxy_config -s -v 
====================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.9.16, pytest-8.0.0, pluggy-1.3.0 -- /data/daver/repos/pinecone-python-client/.venv/bin/python
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /data/daver/repos/pinecone-python-client
plugins: timeout-2.2.0, asyncio-0.15.1, mock-3.6.1, cov-2.10.1, benchmark-4.0.0
collected 6 items                                                                                                                                                                                                                                               

tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_https_proxy_with_self_signed_cert docker run -d --rm --name proxy1 -p 8080:8080 -v /data/daver/repos/pinecone-python-client/tests/integration/proxy_config/.mitm/proxy1:/home/mitmproxy/.mitmproxy mitmproxy/mitmproxy mitmdump
docker run -d --rm --name proxy2 -p 8081:8080 -v /data/daver/repos/pinecone-python-client/tests/integration/proxy_config/.mitm/proxy2:/home/mitmproxy/.mitmproxy mitmproxy/mitmproxy mitmdump --set proxyauth=testuser:testpassword
PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_http_proxy_with_self_signed_cert PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_with_ssl_verification_disabled_emits_warning PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_with_incorrect_cert_path PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_with_valid_path_to_incorrect_cert PASSED
tests/integration/proxy_config/test_proxy_settings.py::TestProxyConfig::test_proxy_that_requires_proxyauth PASSED

======================================================================================================================= 6 passed in 8.47s =======================================================================================================================

@jhamon
Copy link
Collaborator

jhamon commented May 13, 2024

Thanks for following up with this.

I previously put a lot of effort into trying to get tests with mitmproxy running in CI and ultimately had to bail due to time and other urgent things coming up. You can of course run a docker container in github actions, but I kept running into permissions issues when trying to mount a directory as a volume on the mitmproxy container so I could get the generated cert for testing. Details are a bit vague in my memory now, but I think docker was creating files as root that the test processing outside the container could not access. One day I'll get back to it because it's not great having these tests that don't run automtically.

@jhamon jhamon merged commit b174eea into main May 13, 2024
81 checks passed
@jhamon jhamon deleted the daver/grpc_proxy_ssl_ca_cert branch May 13, 2024 18:55
@daverigby
Copy link
Contributor Author

Thanks @jhamon.

Note we probably don't actually need mitmproxy inside a container for this use-case, we could just download and install directly into the GitHub worker (it's a simple tgz containing just 3 binaries - mitmproxy plus the web and dump variants), or it looks like you can even install via pipx: https://docs.mitmproxy.org/stable/overview-installation/

mcpaddy added a commit to mcpaddy/pinecone-python-client that referenced this pull request May 22, 2024
@mcpaddy
Merge branch 'main' of github.com:pinecone-io/pinecone-python-client
96d6994 
* 'main' of github.com:pinecone-io/pinecone-python-client:
  [skip ci] Bump version to v4.1.0
  Bump tqdm from 4.66.1 to 4.66.3 (pinecone-io#344)
  Bump idna from 3.4 to 3.7 (pinecone-io#345)
  Bump jinja2 from 3.1.3 to 3.1.4 (pinecone-io#343)
  Add better error messages for mistaken `from_texts` and `from_documents` (pinecone-io#342)
  Support proxy_url and ssl_ca_certs options for gRPC (pinecone-io#341)
  Remove serverless public preview warnings (pinecone-io#340)
  [skip ci] Bump version to v4.0.0
  Improve upsert throughput by 3x (pinecone-io#334)
  Remove `merge` workflow and update `build-and-publish-docs` workflow to be manually runnable (pinecone-io#335)
  [skip ci] Bump version to v3.2.2
  [Fix] openapi_config deprecation warning incorrectly shown (pinecone-io#327)
  Add grpc unit test run, expand testing of VectorFactory (pinecone-io#326)
  [skip ci] Bump version to v3.2.1
  Allow clients to tag requests with a source_tag (pinecone-io#324)
  [skip ci] Bump version to v3.2.0
  Revise proxy configuration, add integration testing (pinecone-io#325)
  [Fix] Configuring SSL proxy via openapi_config object (pinecone-io#321)
  Update README.md (pinecone-io#323)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhamon jhamon Awaiting requested review from jhamon

@austin-denoble austin-denoble Awaiting requested review from austin-denoble

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@daverigby @jhamon